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Appendix A. About the study 


This appendix describes the general framework, background, and rationale for the study on the progress of non- 
native English speaker students toward English language proficiency and English language arts (ELA) proficiency 
in the early grades in Arizona, including a short review of the literature that shows how the current study builds 
on earlier ones. 


General framework, background, and rationale 


Through the Arizona Literacy Partnership, the Regional Educational Laboratory (REL) West collaborates with the 
Arizona Department of Education (ADE) and local school districts to examine early literacy interventions and 
supports to improve English language proficiency and ELA proficiency by grade 3. The English language proficiency 
and ELA proficiency of English learner students are of specific interest to the partnership. 


This study used existing state data to describe the English language proficiency and ELA proficiency of non-native 
English speaker students in the early grades (kindergarten through grade 3). A goal of the study was to better 
understand how students, especially English learner students, progress in both English language proficiency and 
ELA proficiency in the early grades. This study goes beyond previous analyses by examining how the English 
language outcomes of English learner students and the ELA outcomes of all students, including non-native English 
speaker students not classified as English learner students, vary by student and school characteristics and by 
English language proficiency in kindergarten. 


The results of this study can inform efforts to understand and improve K-3 language and literacy policies and 
programs and can provide important context to support evidence-based decisionmaking for K—3 literacy 
interventions. Specifically, the study findings can help stakeholders better understand the context for early English 
language and ELA progress of non-native English speaker students, including English learner students, and 
patterns of progress. 
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Related literature 


A well-documented, persistent academic achievement gap exists between English learner students and native 
English speaker students (Fry, 2007; Hemphill & Vanneman, 2011). This gap is generally accepted as being related 
to English learner students’ need to simultaneously learn English and master content knowledge (Genesee et al., 
2005). To support non-native English speaker students in achieving English language proficiency, their English 
language proficiency is evaluated at entry into the school system, and language assistance is offered to students 
classified as English learner students. The goal is to improve their English language proficiency and ultimately help 
them master the same academic content and achieve the same standards as other students. 


Student characteristics may contribute to the academic achievement gap for English learner students. Influential 
student characteristics include English language proficiency level on school entry (Cook et al., 2012; Greenberg 
Motamedi et al., 2016; Haas et al., 2015) and demographic characteristics (Haas et al., 2015; Liasidou, 2013). A 
previous REL West study conducted in Arizona (Haas et al., 2015) followed three cohorts of students for six years. 
The study found that the largest differences in cumulative passing rates on the statewide ELA assessment were 
associated with initial English language proficiency level, eligibility for special education services, eligibility for the 
national school lunch program (an indicator of socioeconomic status), and gender. Further, another REL West 
study conducted in Arizona and Nevada examined English learner students’ outcomes after grade 3 and found 
that students who had higher English language proficiency levels in grade 3 had higher passing rates on ELA 
assessments in the subsequent two years (Haas, Tran, & Huang, 2016). However, neither of these studies looked 
at school characteristics and how they might have influenced student outcomes, nor did they examine, as a 
subgroup, non-native English speaker students who demonstrated initial English language proficiency. 


Some school characteristics may influence students’ academic outcomes. Several studies have investigated the 
association between the type of English learner program and students’ academic outcomes (Greenberg Motamedi 
et al., 2016; Parker et al., 2014; Slavin et al., 2011; Umansky & Reardon, 2014). A recent REL Northeast & Islands 
study examined the correlates of academic performance for English learner students in a New England school 
district and found that, in addition to student characteristics, the type of English learner programs in a school was 
closely related to average level of English language proficiency (Parker et al., 2014). The study used cross-sectional 
data, as longitudinal data were not available. A study of elementary school dual language learner students 
(students who are learning English and another language spoken at home), found that those attending larger 
schools with fewer dual language learner students had higher levels of English language proficiency in 
kindergarten and more rapid growth in English language proficiency over time (Kim et al., 2014). This pattern of 
findings merits further study of the association between school characteristics and English language proficiency 
and of the association between school characteristics and ELA outcomes. In addition, previous studies that did 
not focus on English learner students have shown that school composition, such as socioeconomic levels and 
racial/ethnic makeup, matter for early progress in reading achievement (Benson & Borman, 2010; Borman & 
Dowling, 2010). 


Non-native English speaker students who are not classified as English learner students at kindergarten entry are 
an understudied group. While some recent studies have examined the academic performance of different 
subgroups of English learner students (for example, current and former English learner students; Deussen et al., 
2017) and compared their performance with that of non-English learner students, few studies have focused on 
non-native English speaker students who were not classified as English learner students in kindergarten. One study 
used a nationally representative sample of first-time kindergartners from the Early Childhood Longitudinal Study 
to compare outcomes for English and math achievement for non-native English speaker students who were not 
classified as English learner students in kindergarten with outcomes for native English speaker students and 
students classified as English learner students (Halle et al., 2012). The study found that non-native English speaker 
students who were not classified as English learner students in kindergarten kept pace with native English speaker 
students. Among non-native English speakers, entering kindergarten with English language proficiency was 
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associated with better cognitive and behavioral outcomes through grade 8 compared with non-native English 
speakers who attained proficiency after kindergarten entry. 


Building on these earlier studies, the current study used longitudinal data to track a cohort of kindergartners and 
identify student and school characteristics associated with English language proficiency and ELA proficiency in 
grade 3. Specifically, the study examined the student characteristics identified in the 2015 REL West study in 
Arizona as accounting for the largest differences in cumulative passing rates on the statewide ELA assessment 
(Haas et al., 2015). In addition, the current study examined the relationship between school characteristics, such 
as a school’s socioeconomic and language composition (for example, percentage of socioeconomically 
disadvantaged students and percentage of English learner students) and students’ English language proficiency 
and ELA proficiency. The study seeks to enable a better understanding of the association between school 
compositional factors and the development of ELA skills among non-native English speaker students, including 
those who are not classified as English learner students. Furthermore, the English language proficiency assessment 
and reassessment in Arizona (the Arizona English Language Learner Assessment) were developed or revised 
between 2012/13 and 2013/14, and a new state assessment for English language arts in grade 3 (the Arizona 
Measurement of Education Readiness to Inform Teaching) was first administered in spring 2015. Earlier analyses 
used the previous assessments, making it even more important to examine the association between English 
language proficiency and ELA proficiency using current assessments, as in this study. 
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Appendix B. Methods 


This appendix describes the population of analysis, data sources, definitions of measures and variables, analysis 
methods, and multilevel regression analyses for the study. 


Population of analysis 


The study population is based on the cohort of students who entered an Arizona public kindergarten in 2013/14. 
The population of analysis for research question 1 is the cohort of students who entered an Arizona public 
kindergarten in 2013/14 (88,857 students; table B1). For examining the association between English language 
proficiency level in kindergarten and English language proficiency and English language arts (ELA) proficiency levels 
in grade 3, the population of analysis (which is based on the populations of interest to the study’s stakeholders) 
was English learner students (research question 2) or all students (research question 3) who entered an Arizona 
public kindergarten in 2013/14, made regular grade progress (were not retained or held back), were still enrolled 
in the Arizona public school system in 2016/17, and had taken an English language proficiency assessment in 
kindergarten and grade 3 (research question 2) and an ELA proficiency assessment in grade 3 (research question 
3). For research question 2, among the 14,101 students classified as English learner students in kindergarten in 
2013/14 who made regular grade progress and were still enrolled in the Arizona public school system in 2016/17, 
13,776 students (98 percent) had an Arizona English Language Learner Assessment (AZELLA) score available and 
were included in the analysis (see table B1). For research question 3, among the 73,476 students who entered an 
Arizona public kindergarten in 2013/14, made regular grade progress, and were still enrolled in the Arizona public 
school system in 2016/17, 69,654 students (95 percent) had a score on the Arizona Measurement of Education 
Readiness to Inform Teaching (AZMERIT) in grade 3 and were included in the analysis. 


Further descriptions of the characteristics of the population of analysis are provided in tables C2 and C3 in 
appendix C. 


Table B1. Research question and population of analysis, 2013/14 kindergarten cohort in Arizona 


Number of 

Research question exo) ol] F-]ulo)amolar-Var-]Naa(oucy-l ale) (=) students 
1. What percentage of students entering All students entering kindergarten in 2013/14 in Arizona 88,857 
kindergarten in 2013/14 in Arizona were 
classified as English learner students, and 
did the percentage change after 
reassessment at the end of kindergarten? 
2. For students classified as English learner — Students entering kindergarten in 2013/14 in Arizona who were 13,776 
students in kindergarten, how did English classified as English learner students and who met three criteria: 
language proficiency level in grade 3 vary *° Made regular grade progress. 
by English language proficiency level in e Stayed enrolled in the Arizona public school system through 
kindergarten and by the characteristics of 2016/17 (grade 3) 

. . i | " 
ENBISH Earner skugents ald Sele SenoOls! e Had an Arizona English Language Learner Assessment score. 
3. For all students, how did English All students entering kindergarten in 2013/14 in Arizona who met 69,654 


language arts proficiency level in grade 3 three criteria: 
vary by English language proficiency levelin » jyade regular grade progress. 


kindergarten and by the characteristics of e Stayed enrolled in the Arizona public school system through 
students and their schools? 2016/17 (grade 3). 


e Had an Arizona Measurement of Education Readiness to Inform 
Teaching English language arts assessment score. 


Source: Authors’ analysis based on Arizona Department of Education data described in this appendix. 
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Data sources 


The data for this study were obtained from multiple ADE sources. Data on enrollment, English language proficiency 
assessments, and grade 3 ELA assessments were linked to 2013/14 cohort kindergarten students using a unique 
student statewide identification number generated by ADE. This was the dataset of analysis for the study and was 
used to answer the research questions. 


Arizona longitudinal enrollment data system. The Arizona longitudinal enrollment data system provides 
information on students’ demographic characteristics and school enrollment records. Data for this study included 
student records for school years 2013/14—2016/17 for the cohort of students who entered kindergarten in 
2013/14. 


AZELLA. The AZELLA is a standards-based assessment that measures student English language proficiency for 
placement into English learner services and for reassessment of the need for such services. The data for this study 
included students’ AZELLA proficiency levels for the 2013/14 (kindergarten) to 2016/17 (grade 3) school years for 
all students who started kindergarten in the Arizona public school system in 2013/14. At every grade except 
kindergarten, the proficiency levels are pre-emergent/emergent, basic, intermediate, and proficient. The AZELLA 
Kindergarten Placement Test (KPT) uses only three proficiency levels: emergent/emergent, basic/intermediate, 
and proficient. 


AzMERIT. AzZMERIT is the state achievement assessment. Students take AZMERIT content assessments in several 
subjects, including ELA, annually in grades 3-8 and in either grade 10 or 11 in high school. Each AzMERIT 
assessment is aligned to Arizona’s academic standards. AZMERIT was first administered in spring 2015. Data for 
this study included students’ proficiency levels (minimally proficient, partially proficient, proficient, and highly 
proficient) on the 2016/17 ELA assessment in grade 3 for all students who started kindergarten in the Arizona 
public school system in 2013/14. For the multilevel analysis for research question 3, the categories proficient and 
highly proficient were collapsed into a proficient category and the categories minimally proficient and partially 
proficient were collapsed into a below proficient category. 


Public data sources. Schools’ charter school status (whether a school operates under a public charter instead of a 
school district) for all Arizona public schools was downloaded from the public information available on the ADE 


website (https://www.azed.gov/accountability-research/reportcards/). 


Definitions of measures and variables 


Non-native and native English speaker students. Arizona administers the Primary Home Language Other Than 
English (PHLOTE) survey to all students entering kindergarten in the Arizona public school system to identify non- 
native English speaker students. Students whose home language is listed as other than English on any of the three 
questions of the survey (figure B1) are classified as non-native English speaker students. All other students are 
classified as native English speaker students. 
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Figure B1. The Arizona Department of Education’s standard process for identifying non-native English speaker 
students and English learner students in kindergarten 


At enrollment, is the answer to any of the three questions on home language on the Primary Home Language Other Than 
English survey a language other than English? 

1. Primary language used in the home regardless of the language spoken by the student? 

2. Language most often spoken by the student? 

3. Language that the student first acquired? 


Non-native English speaker Native English speaker 
Must be assessed for English language proficiency using Arizona English Not eligible for English learner 
Language Learner Assessment Kindergarten Placement Test services 


Not proficient Proficient 

English learner student Initially fluent English proficient 

¢ — Eligible for placement in English « No further English proficiency 
learner services. assessment. 


Reassessed at the end of the year Not eligible for English learner 
and every year until student services. 
achieves English proficiency. 


Note: This standard process was not followed in 2013/14 because nearly all non-native English speaker students in kindergarten were reassessed at the end 
of the year as part of a standard-setting review process (Arizona Department of Education, 2014). 
Source: Authors’ analysis based on Arizona Department of Education data described in this appendix. 


English language proficiency measures. As previously described, the AZELLA is Arizona’s assessment of English 
language proficiency. It is aligned to the Arizona English Language Proficiency Standards and is administered to 
students who are identified by the PHLOTE survey as non-native English speaker students (Arizona Department of 
Education, 2014). The AZELLA includes the Kindergarten Placement Test (KPT) to determine whether a non-native 
English speaker needs English learner services and an annual reassessment to monitor the progress of English 
learner students until they achieve English language proficiency. According to Arizona Revised Statutes 15-752, 
proficiency is attained when “English learners have acquired a good working knowledge of English and are able to 
do regular school work in English” (Arizona Department of Education, 2014). Students who score below proficient 
on the AZELLA are considered not to have sufficient knowledge of English to do regular schoolwork in English. 
They are classified as English learner students and become eligible for English learner services. 


Scores on the AZELLA KPT, administered at the beginning of the kindergarten school year, are reported at three 
proficiency levels: pre-emergent/emergent, basic/intermediate, and proficient (table B2). 
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Table B2. The three proficiency levels on the Arizona English Language Learner Assessment Kindergarten 
Placement Test 


Proficiency level Description 


Pre-emergent/emergent Students at this level lack the English skills to communicate, retell stories heard, or add details to 
drawings. These students do not demonstrate sufficient skills in English to access mainstream 
curriculum, and they demonstrate the need for specific support in English language development 
instruction. 


Basic/intermediate Students at this level generally understand spoken English but do not have the vocabulary to 
respond consistently. They orally communicate basic needs and ideas with gestures and isolated 
English words. They use pictures to recall objects from a story heard, repeat words that begin with 
the same first sound, and add minimal details to drawings. These students do not demonstrate 
sufficient skills in English to access mainstream curriculum, and they demonstrate the need for 
specific support in English language development instruction. 


Proficient Students at this level listen and respond appropriately to spoken English. They have an expanded 
English vocabulary to orally communicate basic needs and ideas with English words, phrases, and 
sentences with correct pronunciation. They use pictures or words to retell events from a story 
heard, identify pictures with the same first sound, and add relevant details to drawings. These 
students demonstrate the skills necessary to access mainstream curriculum. 


Source: Arizona Department of Education (https://cms.azed.gov/home/GetDocumentFile?id=5852d396aadebe0658611fd0). 


Scores on the AZELLA Reassessments are reported at four levels of English language proficiency: 
pre-emergent/emergent, basic, intermediate, and proficient. Students who are classified as English learner 
students take the AZELLA Reassessment annually until they demonstrate English language proficiency. 


These AZELLA proficiency levels are the measures of English language proficiency used in this study. Students who 
demonstrate English language proficiency on the AZELLA are reclassified as proficient and are no longer eligible 
for English learner services. 


The 2013/14 AZELLA standard setting review. As part of a voluntary resolution agreement with the federal 
government related to a complaint filed against ADE on how English learner students were reclassified as English 
language proficient and no longer eligible for English learner services (U.S. Department of Education, 2012; U.S. 
Department of Justice, 2012), ADE developed the AZELLA KPT in 2012/13. In spring 2014 ADE convened a Technical 
Advisory Committee to define cutscores for the AZELLA KPT categories of basic/intermediate and proficient.’ The 
committee recommended higher cutscores for the two categories for school year 2014/15. During this standard- 
setting review, non-native English speaker students, including students initially classified as proficient in English, 
were reassessed. Typically, as outlined in figure B1, students who score proficient on the AZELLA KPT receive no 
other assessment of their English language proficiency during the same school year (unless referred by a teacher). 
However, as part of the standard-setting review, 95 percent of non-native English speaker students in the 2013/14 
kindergarten cohort were systematically reassessed during kindergarten using the AZELLA Stage | Reassessment. 


English learner students as of kindergarten. Students who demonstrate an English language need on the AZELLA 
KPT immediately qualify for English learner services. Therefore, according to ADE policy, English learner students 
include all students evaluated as having an English language need during their kindergarten school year. 
Kindergarten students who demonstrate an English language need include: 


e Students classified as English learner students on the AZELLA KPT. 


1 Further information about the 2013/14 standard-setting review can be found in Arizona Department of Education (2014). 
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e Students who were either assessed as proficient on the AZELLA KPT or who were not assessed based on the 
PHLOTE survey but who subsequently qualified for English language services during kindergarten based on 
the AZELLA Stage | Reassessment. 


Confirmed as proficient in kindergarten. For the purposes of this study non-native English speaker students initially 
classified as English language proficient on the AZELLA KPT and confirmed as proficient at the AZELLA Stage | 
Reassessment at the end of kindergarten are referred to as students confirmed as proficient in kindergarten. 


Reclassified as English learner students in kindergarten. For the purposes of this study non-native English speaker 
students initially classified as English language proficient based on the AZELLA KPT but who scored below proficient 
on the AZELLA Stage | Reassessment at the end of kindergarten and were therefore reclassified as English learner 
students are referred to as reclassified as English learner students in kindergarten. 


English language proficiency subgroups. English language proficiency subgroups were constructed to follow 
student progress in English skills through grade 3 and to measure the association between English language 
proficiency level in kindergarten and ELA proficiency in grade 3. A student’s initial English language proficiency 
level was defined as of the first AZELLA assessment in kindergarten that qualified the student for English learner 
services. The following subgroups were defined based on the definitions of English language proficiency: 


e Native English speaker students. 


e Students confirmed as proficient in kindergarten (initially classified as English language proficient and 
confirmed as proficient at the end of kindergarten). 


e Reclassified as English learner students in kindergarten (initially classified as English language proficient and 
reclassified as English learner students at the end of kindergarten). 


e Other English learner students who scored at the basic/intermediate level on their first AZELLA assessment. 


e Other English learner students who scored at the pre-emergent/emergent level on their first AZELLA 
assessment. 


The distribution of students across these English language proficiency subgroups is shown in table C1 in 
appendix C. 


ELA proficiency. ELA proficiency was measured by the AZMERIT ELA assessment in grade 3, administered to all 
students, and was examined across the English language proficiency subgroups defined above. A student’s ELA 
performance is reported as being at one of four proficiency levels: minimally proficient, partially proficient, 
proficient, or highly proficient. Students who score minimally proficient or partially proficient are considered likely 
to need support to be ready for the next grade or course, and those who score proficient or highly proficient are 
considered likely to be ready for the next grade or course. 


Student characteristics. Based on previous related literature that identified certain student characteristics as 
potentially associated with the academic outcomes of non-native English speaker students (see appendix A), the 
study team included in the analysis the following student characteristics for which data were available in the ADE 
datasets: 


e Socioeconomic status (as indicated by eligibility for the national school lunch program). 
e Gender. 

e Race/ethnicity. 

e Eligibility for special education services. 


These variables were defined as of entry into kindergarten. 


REL 2020-024 B-5 


School characteristics. Based on previous related literature that identified certain school characteristics as 
potentially associated with the academic outcomes of non-native English speaker students (see appendix A), the 
study team included in the analysis the following school characteristics for which data were available in the ADE 
datasets: 


e Percentage of students from low socioeconomic status families (as indicated by eligibility for the national 
school lunch program). 


e Percentage of English learner students. 
e School size. 


Those variables were derived from student-level records from the Arizona longitudinal enrollment database for 
the cohort of analysis at entry into kindergarten in 2013/14. For example, school size is approximated by the size 
of the 2013/14 kindergarten cohort in the school, and the percentage of English learner students is computed 
specifically for the cohort of analysis rather than schoolwide. In addition, because Arizona has a high percentage 
of students enrolled in charter schools (18 percent in 2013/14, according to U.S. Department of Education, 2015), 
schools’ charter status was included. 


Analysis methods 


This study used different quantitative methods of analysis for each research question. 


Research question 1. What percentage of students entering kindergarten in 2013/14 in Arizona were classified as 
English learner students, and did the percentage change after reassessment at the end of kindergarten? The study 
team calculated descriptive statistics on the number and percentage of students at each English language 
proficiency level at kindergarten entry as measured by the AZELLA KPT (pre-emergent/emergent, 
basic/intermediate, and proficient) and after reassessment at the end of kindergarten. 


Research question 2. For students classified as English learner students in kindergarten, how did English language 
proficiency level in grade 3 vary by English language proficiency level in kindergarten and by the characteristics of 
English learner students and their schools? The study team first calculated descriptive statistics on the distribution 
of English language proficiency levels in kindergarten by English language proficiency level in grade 3 for students 
classified at the pre-emergent/emergent level, students classified at the basic/intermediate level, and students 
reclassified as English learner students in kindergarten. Starting with the kindergarten AZELLA Stage | 
Reassessment (administered at the end of kindergarten), proficiency levels were reported using four categories: 
pre-emergent/emergent, basic, intermediate, and proficient. Students scoring at the proficient level on the 
AZELLA Reassessment were reclassified as proficient for the following school year. A chi-squared test of the 
independence of English language proficiency levels in kindergarten and grade 3 was used to estimate whether 
the distribution of English language proficiency levels at the end of grade 3 varied by students’ English language 
proficiency level in kindergarten. Independence was rejected (X? [6, N = 13,776] = 392, p< .0001). 


Next, the study team conducted a regression analysis that included student characteristics (English language 
proficiency level in kindergarten, gender, eligibility for special education services, socioeconomic status, and 
race/ethnicity) and school-level characteristics (percentage of economically disadvantaged students, percentage 
of English learner students, charter school status, and school size). These characteristics were selected because 
data were available from ADE and because previous research had found them to be correlated with English 
language proficiency. Further details on the analysis method are provided in the section below on multilevel 
regression analyses. 


Research question 3. For all students how did ELA proficiency level in grade 3 vary by English language proficiency 
level in kindergarten and by the characteristics of students and their schools? The study team first calculated 
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descriptive statistics on the distribution of ELA proficiency levels on the AZMERIT ELA assessment in grade 3 by 
English language proficiency level in kindergarten. Specifically, the percentages of students who scored minimally 
proficient, partially proficient, proficient, and highly proficient on the ELA assessment in grade 3 were calculated 
for non-native English speaker students by their English language proficiency level in kindergarten (pre- 
emergent/emergent, basic/intermediate, reclassified as English learner student in kindergarten, and confirmed 
as proficient in kindergarten). The percentage of native English speaker students at each ELA proficiency level in 
grade 3 was also calculated as a comparison distribution. A chi-squared test of the independence of ELA 
proficiency level in grade 3 and English language proficiency level in kindergarten among non-native English 
speaker students was used to estimate whether the distribution of ELA proficiency levels in grade 3 varied by 
students’ English language proficiency level in kindergarten. Independence was rejected (X? [6, N = 19,028] = 3312, 
p< .0001). 


Next, the study team conducted a multilevel regression analysis using ELA proficiency level in grade 3 as the 
dependent variable and the same student and school characteristics used in research question 2 to identify 
characteristics correlated with non-native English speaker students’ ELA proficiency level in grade 3. Further 
details on the analysis method are provided in the next section on multilevel regression analyses. 


Multilevel regression analyses 


Because students are nested within schools, the study team used a random intercept multilevel regression model 
(O’Dwyer & Parker, 2014; Raudenbush & Bryk, 2002; Snijders & Bosker, 1999) to examine which student and 
school characteristics were most closely associated with achieving English language proficiency and ELA 
proficiency by the end of grade 3. 


Models. Because the dependent variable was dichotomous for research questions 2 and 3, hierarchical generalized 
linear models, allowing for binary, non-normally distributed responses, were estimated, and the dependent 
variable was transformed using the logit link function (Ene et al., 2015; Luke, 2004). The random intercept model 
accounting for individual students (level 1) nested within schools (level 2) can be represented as: 


(B1) — Zj= log [Pi/ (1- Pj) 
Zi = Boj + BiXai + B2X oii +. + BrXnij (level 1) 


where Z;, the dependent variable, is the log odds of achieving English language proficiency by the end of grade 3 
or the log odds of achieving ELA proficiency by the end of grade 3 for student j in school j; P; is the probability of 
achieving English language proficiency or the probability of achieving ELA proficiency for student ij in school j; Xnij 
is a set of student characteristics (English language proficiency level in kindergarten and demographic 
characteristics); Bo; represents a school j constant; and 8B, is a regression coefficient indicating the predicted 
change in Z; for every one-unit increase in the value of the associated X, variable, holding constant the other 
variables in the model. 


At the school level the random intercept model can be represented as: 
(B2) Boj = Yoo + YorW1j + Yo2Waj+ ... + YonWrj + Uoj (level 2) 


where Yo,W,, represents a set of school characteristics, such as school demographic composition and charter 
school designation, and ugis a school error term that is assumed to be normally distributed with a mean of zero 
Ugj~ N(0,Too). Each predictor was centered to its grand mean (the average proportion of each demographic in the 
analytic sample). Therefore, the estimated intercept Yoo represents the log odds of achieving English language 
proficiency and ELA proficiency for a typical student at a typical school by the end of grade 3. The models were 
estimated using the SAS GLIMMIX procedure with the Laplace estimation method. 
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Intraclass correlation coefficient and variance partition. For each dependent variable the study team first 
estimated an unconditional multilevel model (a model without any predictors and including only a random school 
effect). These unconditional models were used to partition the variance of the dependent variable into within- 
school and between-school variance. The intraclass correlation coefficient (ICC) for each unconditional regression 
calculated the portion of variance that was due to between-school differences: 


(B3) ICC = T299 / (T2o0+ 3.29). 


If there is no statistical dependency due to the nesting of students within schools, all of the variance would be 
expected to be among students, and the ICC would be close to zero. In contrast, with highly dependent data the 
largest proportion of variance would be between schools, and the ICC would be closer to 1 (O’Dwyer & Parker, 
2014). In both regression analyses the school random effect was significant, but the percentage of variance due 
to between-school differences was relatively low: the ICC was 6.5 percent for achieving English language 
proficiency and 16.4 percent for achieving ELA proficiency, indicating that most of the variation was due to within- 
school differences or to unknown factors (see tables C6 and C10 in appendix C). Because the effect was significant, 
a school random intercept was retained in the analyses, but no further attempt was made to include school 
random effects. 


The study team then added to the model student- and school-level fixed effects of interest (predicted probabilities 
are presented in tables 1 and 2 in the main report). To facilitate the interpretation of the findings, the study team 
provided odds ratios and predicted probabilities in addition to the estimated coefficients (see tables C7 and C11 
in appendix C). 


Odds ratio. An odds ratio (OR) is a relative measure of effect, which enables comparing the effect for one group 
of a study relative to the effect for a comparison or reference group. The odds ratio of group 1 relative to group 2 
is defined as: 


(B4) — OR12= OddSgroup1 | OddSgroup2 = [P1/ (1 — P1)] / [P2/ (1 — P2)] 


where P, is the probability of achieving English language proficiency or ELA proficiency for group 1 and P2 is the 
probability of achieving English language proficiency or ELA proficiency for group 2. 


For each variable, a category with an odds ratio greater than 1 means that students in that category were more 
likely to achieve English language proficiency or to achieve ELA proficiency than students in the reference 
category. A category with an odds ratio less than 1 means that students in that category had lower odds of the 
dependent variable than students in the reference category. 


For the model described above, with group 2 as the reference category, the group 1 coefficient (B1) is an estimate 
of the log odds ratio of group 1 versus group 2. Therefore, a significant coefficient B: indicates a significant 
difference between the two groups, and OR:2 can be estimated as exp (B:). 


Predicted probabilities. Predicted probabilities were computed by applying the inverse link function to the least 
squares means of each group effect on the logit scale, with all other student and school variables at the average 
value for the population.? For example, the predicted probability for native English speaker students was 
computed for an average student (other than being classified as a native English speaker) attending an average 
school. This predicted probability can be compared with the predicted probability for students confirmed as 
proficient in kindergarten, which was computed for an average student (other than being confirmed as proficient 
in kindergarten) attending an average school. 


2 2/3 =3.29 is the variance of the standard logistic distribution (Snijders & Bosker, 1999). 
3 Using GLIMMIX LSMEANS statement. 
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Appendix C. Supporting analyses for research questions 2 and 3 


Appendix C provides additional detailed results, including summary information about the population of analysis 
and estimated coefficients, odds ratios, and predicted probabilities for the multilevel regression analyses 
estimated for research questions 2 and 3. 


Table C1. Distribution of native and non-native English speaker students at kindergarten entry, 2013/14 
kindergarten cohort in Arizona 
Student group and English language 


proficiency level at kindergarten entry Number Percent 
Native English speaker students 66,756 75.1 
Non-native English speaker students 22,101 24.9 
Proficient 11,940 13.4 
Nonproficient 10,161 11.4 
Basic/intermediate 4,987 5.6 
Pre-emergent/emergent 5,174 5.8 
Total 88,857 100.0 


Note: Percentages may not sum to totals because of rounding. 
Source: Authors’ analysis based on Arizona Department of Education data described in appendix B. 
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Table C2. The number and percentage of students for the categorical variables of analysis, 2013/14 
kindergarten cohort in Arizona 


Student characteristic Number Percent 
Total 2013/14 kindergarten cohort 88,857 100.0 
Total English learner students 16,257 18.4 
English language proficiency level in kindergarten 
Non-native English speaker students 22,837 25.7 
Pre-emergent/emergent 5,392 6.1 
Basic/intermediate 5,219 5.9 
Reclassified as English learner student in kindergarten 5,646 6.4 
Confirmed as proficient in kindergarten 6,590 7.4 
Native English speaker students 66,010 74.3 
Gender 
Female 43,514 49.0 
Male 45,343 51.0 
Eligible for special education services 
No 80,750 90.9 
Yes 8,107 9.1 
Economically disadvantaged status 
No 46,331 52.1 
Yes 42,526 47.9 
Race/ethnicity 
American Indian or Alaska Native 4,132 4.7 
Asian 2,295 2.6 
Black/African American 4,328 4.9 
Hispanic/Latino 41,380 46.6 
White 33,402 37.6 
Other 3,320 3.7 
Charter school status? 
Yes 14,723 16.6 
No 74,072 83.4 


a. Charter status of school of enrollment in kindergarten was missing for 62 students. 
Source: Authors’ analysis based on Arizona Department of Education data described in appendix B. 


Table C3. Distribution of students by school characteristics for the continuous variables of analysis, 2013/14 
kindergarten cohort in Arizona 


Total number of 


2013/14 Syielater-lae| 
Yel aYole) ol a¥-ler-lei kasi (OM-Var-] el =) ialatel=1e-x-]atal=16s Mean deviation Minimum Maximum 
Percentage of English learner students 88,857 18.3 18.946 0 100 
Number of students 88,857 100.1 45.943 1 297 
Percentage of economically disadvantaged students 88,857 47.9 32.306 0 100 


Note: Variables are computed for the cohort of analysis. Number of students in school is approximated by the number of students in the cohort. Some very 
small schools in Arizona have fewer than 10 students in the elementary grades. 
Source: Authors’ analysis based on Arizona Department of Education data described in appendix B. 
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Table C4. Number of English learner students by English language proficiency levels in kindergarten and in 
grade 3, 2013/14 kindergarten cohort in Arizona 
English language proficiency level in kindergarten 


Reclassified as 


English learner 


Pre-emergent/ Basic/ student in 
English language proficiency level in grade 3 emergent intermediate kindergarten 
Pre-emergent/emergent 236 95 65 396 
Basic 931 633 662 2,226 
Intermediate 865 823 1,126 2,814 
Reclassified as English learner student in kindergarten 2,178 2,899 3,263 8,340 
Total 4,210 4,450 5,116 13,776 


Note: The total of 13,776 English learner students includes students who made regular grade progress and were still enrolled in the Arizona public school 
system in 2016/17. 
Source: Authors’ analysis based on Arizona Department of Education data described in appendix B. 


Table C5. Percentage of English learner students by English language proficiency levels in kindergarten and in 
grade 3, 2013/14 kindergarten cohort in Arizona 
English language proficiency level in kindergarten 
Reclassified as 


English learner 


Pre-emergent/ Basic/ student in 
English language proficiency level in grade 3 emergent intermediate alatel=eex-l ania 
Pre-emergent/emergent 5.6 2.1 1.3 
Basic 22.1 14.2 12.9 
Intermediate 20.5 18.5 22.0 
Reclassified as English learner in kindergarten 51.7 65.1 63.8 
Total 100 100 100 


Note: A total of 13,776 English learner students made regular grade progress and were still enrolled in the Arizona public school system in 2016/17. 
Percentages may not sum to 100 because of rounding. 
Source: Authors’ analysis based on Arizona Department of Education data described in appendix B. 


Table C6. Intraclass correlation coefficient and variance partition for an unconditional multilevel model of 
achieving English language proficiency by the end of grade 3, 2013/14 kindergarten cohort in Arizona 
Intraclass 


correlation 
(oN 7- Jar alers) coefficient 


parameter Subject Estimate Si-lalol-lncMlanelg Z-value (percent) 
Too School 0.2275 0.03035 7.5 < .0001 6.5 


Note: The unconditional model with random intercept can be written as Zj = Boj + Uoj with Uoj~ N(O, Too). ICC (intraclass correlation coefficient) = T7o0 / 
(t?00 + 3.29). Estimates obtained with SAS GLIMMIX with a population of 13,776 students who were classified as English learner students in kindergarten, 
made regular grade progress, and were still enrolled in the Arizona public school system in 2016/17. 

Source: Authors’ analysis based on Arizona Department of Education data described in appendix B. 
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Table C7. Predicted probability of achieving English language proficiency by the end of grade 3, by student and 
school characteristics, 2013/14 kindergarten cohort in Arizona 


Estimated Predicted 
coefficient (@fololme-lile) probability 
Characteristic (Giiclalerlacm-lacela) estimate (percent) 
Intercept (average student) 0.479** 61.8 
(0.0322) 


Student characteristic 


English language proficiency in kindergarten 


Pre-emergent/emergent —0.641** 0.53 50.9 
(0.0474) 

Reclassified as English learner student in kindergarten —0.001 1.00 66.3 
(0.0466) 

Basic/intermediate? 66.3 

Gender 

Female 0.238** 1.27 64.6 
(0.0373) 

Male? 59.0 


Eligible for special education services 


Yes —0.945** 0.39 40.5 
(0.0678) 
No? 63.6 


Economically disadvantaged status 


Yes —0.005 1.00 61.7 
(0.0634) 
No? 61.8 


Race/ethnicity 


American Indian or Alaska Native —0.911** 0.40 38.7 
(0.1686) 

Asian 0.742** 2.10 76.7 
(0.1236) 

Black/African American 0.230 1.26 66.4 
(0.1689) 

Hispanic/Latino 61.1 

White 0.549** 1.73 73.1 
(0.1255) 

Other 0.297 1.35 67.8 
(0.2977) 
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Estimated Predicted 


coefficient (Ofoolma-lile) probability 
Characteristic (Giclarerlaem-lacela) estimate (percent) 


School characteristic 


Percentage of English learner students 


Higher than average by 10 percentage points —0.005 1.00 61.6 
(0.0152) 

Number of students 

Higher than average by 20 students 0.044** 1.04 62.8 
(0.0068) 


Percentage of economically disadvantaged students 


Higher than average by 10 points —0.008 0.99 61.6 
(0.0111) 
Charter school status 


Yes —0.004 1.00 61.7 
(0.0894) 
No? 61.8 


** Significant at p = .01. 

a. Reference category for the categorical predictors. 

Note: No additional probabilities were significant at p = .05, and no probabilities were significant at p = .001. 
Source: Authors’ analysis based on Arizona Department of Education data described in appendix B. 


Table C8. Number of students by English language proficiency level in kindergarten and English language arts 
proficiency level in grade 3, 2013/14 kindergarten cohort in Arizona 


English language arts proficiency level in grade 3? 


Minimally Partially Highly 
English language proficiency level in kindergarten proficient proficient Proficient proficient Total 
Total 29,350 8,720 22,001 9,583 69,654 
‘Engishlearnerstudents 
Pre-emergent/emergent 2,950 422 590 91 4,053 
Basic/intermediate 2,854 542 831 115 4,342 
“Reclassified as English learnerinkindergarten 3,121 721 1,032 101 4,975 
Students proficient in English language in kindergarten 
“NativeEnglishspeaker = sti‘; é«887300—~— 6203) 17,2220 8,328 ~~ 50,626 
Confirmed as proficient in kindergarten 1,552 832 2,326 948 5,658 


a. As assessed by the Arizona Measurement of Education Readiness to Inform Teaching. 
Source: Authors’ analysis based on Arizona Department of Education data described in appendix B. 


Table C9. Percentage of students by initial English language proficiency level in kindergarten and English 
language arts proficiency level in grade 3, 2013/14 kindergarten cohort in Arizona 


English language arts proficiency level in grade 3? 


Minimally Partially Highly 
English language proficiency level in kindergarten re)geyitel (lays proficient Proficient proficient a Ko} -}| 


English learner students 


“Pre-emergent/emergent = = 728 104 146 23 100.0 — 

“Basic/intermediate = = = 657 125 191 27 1000 — 
Reclassified as English learner student in kindergarten 62.7 14.5 20.7 2.0 100.0 
Students proficient in English language in kindergarten 

“NativeEnglish speaker = i ti(itsti‘C;;!!!O!!!!O( 23! (U2) lO) 165 ~~ 1000S 

“Confirmedas proficientinkindergarten = §= 24 147 411 168 1000 | 


a. As assessed by the Arizona Measurement of Education Readiness to Inform Teaching. 
Note: Percentages may not sum to 100 because of rounding. 
Source: Authors’ analysis based on Arizona Department of Education data described in appendix B. 


Table C10. Intraclass correlation coefficient and variance partition for an unconditional multilevel model of 
achieving English language arts proficiency by the end of grade 3 


Covariance (ee 
parameter Subject Estimate S\i-lalol-lncW-lanelg VENT) P,>Z (percent) 
Too School 0.6465 0.03255 19.86 < .0001 16.4 


Note: The unconditional model with random intercept can be written as Zj = Boj + Uo with Uoj~ N(0,To0). ICC (intraclass correlation coefficient) = T2o0 / 
(t200 + 3.29). Estimates obtained with SAS GLIMMIX with a population of 69,654 students who made regular grade progress, were still enrolled in the Arizona 
public school system in 2016/17, and had an Arizona Measurement of Education Readiness to Inform Teaching English language arts assessment proficiency 
level. 

Source: Authors’ analysis based on Arizona Department of Education data described in appendix B. 


Table C11. Predicted probability of achieving English language arts proficiency in grade 3, by student and 
school characteristics, 2013/14 kindergarten cohort in Arizona 


Estimated Odds Predicted 
coefficient ratio probability 
Characteristic (standard error) estimate (percent) 
Intercept (average student) —0.24** 44.0 
(0.018) 


Student characteristic 


English language proficiency level in kindergarten 


Pre-emergent/emergent —0.98** 0.38 24.3 
(0.049) 

Basic/intermediate —0.66** 0.52 30.6 
(0.043) 

Reclassified as English learner in kindergarten —0.68** 0.51 30.3 
(0.039) 

Confirmed as proficient in kindergarten 0.73** 2.08 64.0 
(0.033) 


Native English speaker student? 46.1 


Estimated 


Odds 
ratio 
estimate 


Predicted 
probability 


coefficient 

Characteristic (Girclalerlacm-lanela) 

Gender 

Female 0.23** 
(0.017) 

Male? 

Eligible for special education services 

Yes —0.90** 
(0.035) 

No? 

Economically disadvantaged status 

Yes —0.52** 
(0.022) 

No? 

Race/ethnicity 

American Indian or Alaska Native —0.81** 
(0.058) 

Asian 0.92** 
(0.063) 

Black/African American —0.18** 
(0.043) 

Other 0.28** 
(0.046) 

White 0.48** 
(0.023) 

Hispanic or Latino ® 

School characteristic 

Percentage of English learner students 

Higher than average by 10 percentage points —0.11** 
(0.010) 

Number of students 

Higher than average by 20 students 0.01 
(0.008) 

Percentage of economically disadvantaged students 

Higher than average by 10 points —0.04** 
(0.006) 

Charter school status 

Yes 0.0009 
(0.043) 


No? 


** Significant at p = .01. 

a. Reference category for the categorical predictors. 

Note: No additional probabilities were significant at p = .05, and no probabilities were significant at p = .001. 
Source: Authors’ analysis based on Arizona Department of Education data described in appendix B. 
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Appendix D. Analysis of the number of years classified as English learner students 


This appendix describes how the study team estimated the number of years that students in each group of English 
learner students in the 2013/24 kindergarten cohort in Arizona were classified as English learner students, on 
average (table D1). This analysis provides additional information showing that the supports that English learner 
students received over the years were not enough to enable them to achieve English language arts (ELA) 
proficiency in grade 3. 


Table D1. Average number of years students were classified as English learner students by the end of grade 3, 
by English language proficiency level in kindergarten, 2013/14 kindergarten cohort in Arizona 
Reclassified as 
English learner 


Pre-emergent/ Basic/ student in 
S\icliiile emergent Taikclaantsvelr-1a-) kindergarten? 


Number of years classified as English learner student, 


2. 2.61 2.1 
on average, by the end of grade 3 = ? 


Number of students 4,420 4,565 5,116 


a. The kindergarten year was not counted as a year of classification because most of these students were reclassified at the end of the school year and were 
not eligible for services at the beginning of the year. 
Source: Authors’ analysis based on Arizona Department of Education data described in appendix B. 


The number of years students in the 2013/14 kindergarten cohort were classified as English learner students was 
estimated based on their English learner status at the end of each school year. For kindergarten students initially 
classified as English language proficient and reclassified as English learner students at the end of kindergarten, the 
kindergarten year was not counted as a year of classification because most of these students were reclassified at 
the end of the school year and were not eligible for English learner services at the beginning of the year. The 
number of years classified as an English learner student only approximates the number of years of English learner 
services received because families can refuse services and because students who are reclassified in the middle of 
the year typically receive only half a year of services. 


Because students reclassified as English learner students in kindergarten were not eligible for English learner 
services at the beginning of kindergarten, these students tended to be classified as English learner students for 
the fewest number of years from kindergarten through grade 3. These students were classified as English learner 
students for, on average, 2.15 years, suggesting that they achieved English language proficiency and ended 
eligibility for English learner services around grade 2. Students classified at the basic/intermediate level of English 
language proficiency in kindergarten ended eligibility for English learner services after 2.61 years (during grade 2), 
and students classified at the pre-emergent/emergent level in kindergarten ended eligibility for English learner 
services after nearly 3 years (during grade 3). 


A follow-up analysis could look more closely into students’ performance growth in English language and ELA by 
the type of services received and by the length of exposure to those services. 
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